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ABSTRACT 
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generated by a unidimensional and locally independent model. Revised 
Modified Parallel Analysis (RMPA) generalizes MPA and alleviates some 
of its technical limitations. An important and useful feature is a 
new method for eliminating items which violate the test's 
unidimensionality. This is achieved by eliminating items, one at a 
time, to determine their contribution to the matrices* eigenvalues. 
We propose a test for detecting items with larger impact in the 
observed data set, and eliminating them. The new method was tested in 
several situations in which unidimensional item pools were 
"contaminated" by various proportions of items from a- secondary pool. 
The results indicate that RMPA does an excellent job in detecting low 
(10%) and moderate (25%) levels of contamination, but fails in cases 
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A REVISED MODIFIED PARALLEL ANALYSIS (RMPA) FOR 
THE CONSTRUCTION OF UNIDIMENSIONAL ITEM POOLS 



ABSTRACT 



Modified Parallel Analysis (MPA) is a heuristic method for assessing "approximate 
unidimensionality" of item pools. It compares the second eigenvalue of the observed 
correlation matrix with the corresponding eigenvalue extracted from a "parallel" matrix 
generated by a unidimensional and locally independent model. 

Revised Modified Parallel Analysis (RMPA) generalizes >^A and alleviates some of its 
technical limitations. An important and useful feature is a new method for eliminating items 
which violate the test's unidimensionality. This is achieved by eliminating items, one at a 
time, to determine their contribution to the matrices' eigenvalues. 

We propose a test for detecting items with larger impact in the observed data set, and 
eliminating them. The new method was tested in several simulations in which unidimensional 
item pools were "contaminated" by various proportions of items from a secondary pool. The 
results indicate that RMPA does an excellent job in detecting low (10%) and moderate (25%^ 
levels of contamination, but fails in cases of maximal (50%) contamination. 
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A REVISED MODIFIED PARALLEL ANALYSIS (RMPA) FOR THJ 
CONSTRUCTION OF UNIDIMENSIONAL ITEM POOLS 

The increasing popularity of Item Response Theory (IRT) (e.g. Hambleton, 1983; Huhn, 
Drasgow & Parsons 1983; Lord, 1980) in educational, personnel and psychological testing 
has caused a revolution in this domain. Though multidimensional item response models have 
been developed (e.g. Reckase, 1985; Sympson. 1978), most readily applicable IRT models 
used today assume that the test takers responses to all items depend on a single latent trait 
(ability). Thus, it is crucial to establish that any item used in estimating the examinee's 
position along this ability continuum measures, in fact, the same trait. In other words, the 
need to demonstrate that a given item pool is truly unidimensional is a necessary condition for 
many applications of IRT. 

Defining and Assessing Unidimensionality 

Consider a test consisting of n items selected from a larger item pool. Ut ^\ be the vector of n 
binary responses to the test's items (taking values of 1 and 0 for correct and incorrect 
response, respectively), generated by the ith test taker (i=l...N), and let U^. be her response 
to the jth item (j=l -n). Finally, let Gj be a vector of t latent traits characterizing the 
examinee's abiUties. The strong principle of local independence (McDonald, 1981) states that: 

P(Ui= uiiei)= n Pj(Uij= uijlSi) (I) 
j=i 

This principle asserts that the responses to any pair of items are statistically mutually 
independent for any individual, or any subpopulation with fixed latent uraits. The 
dimensionality of U is, simply, the minimal number of latent traits necessary to produce a 
(strong) locally independent model for U. Thus, a pool is umdimensional if responses to all its 
items can be produced by unidimensional locally independent models. 

Although a voluminous literature exists on the issue of unidimensionality of items and tests 
(see Bcrgcr and Knol, 1990; Hattie, 1984 and 1985 for partial reviews), currently there is no 
single approach which is fully satisfactory and/or universally accepted. Hattie (1984) 
compiled a list of 87 measures of unidimensionality and classified them into five 
nonoverlapping classes according to their underlying rationale. He distinguished between 
indices based on 
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(L) closeness to specific answer patterns, 

(ii) reliability coefficients, 

(iii) principal components (PC), 

(iv) factor analysis (FA) and 

(v) goodness of fit to various IRT models. 

Hatlie questioned the theoretical rationale of indices based on response patterns and reliability 
and showed empirically that the measures based on PC, FA and one parameter IRT (the Rasch 
model) are outperformed by methods quantifying deviation from multi-parameter IRT models. 



"Approximate " Unidimensionality 

Many researchers have argued, based on theoretical and empirical observations, that purely 
unidimensional tests, or pools, are quite rare ( e.g. Ackerman, 1989; Humphreys. 1985; 
Reckase, Ackerman & Carlson, 1988; Traub, 1983; Yen, 1984, 1985). If, in fact, 
unidimensionality is frequently violated it is important to determine the practical implications of 
buch violations. Following Reckase's original work (1979), several researchers (e.g. 
Orasgow & Parson, 1983; Yen, 1984, 1985) have shown that unidimensional models are 
quite robust under multidimensionality as long as there is a single "dominant" factor, and 
item difficulty is not confounded with dimensionality. 

These, and other similar, studies suggest that strict unidimensional pools are not necessary for 
many practical applications of unidimensional IRT models (e.g. CAT). It is, however, 
important to develop methods that can identify pools which deviate from strict 
unidimensionaUty to a degree which docs not seriously affect the fit or accuracy of the 
unidimeosionai IRT model. 

This is the motivation behind recent work by Stout, who developed a test of the essential 
unidimensionality o a data set (Stout, 1987,1990; Nandakumar, 1991). Essential 
independence is achieved if the mean covariance (conditional on 8i, the test takei^s vector of i 
latent traits) between all n(n-l)/2 pairs of items approaches 0 as the number of items increases 
to infinity, and the essential dimensionaUty of a pool is the smallest number of latent traits 
necessary to satisfy essential independence. Essential independence is a weaker requirement 
than strong local independence and, in practice, it is obtained whenever there is a single 
dominant dimension in the daU (e.g. Nandakumar, 1991). 
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In the same spirit Drasgow and Lissak(1983) presented Modified Parallel Analysis (MPA for 
short) as "a technique that can determine when an item pool is sufficiently unidimensional for 
the use of IRT" (Drasgow and Lissak. 1983, page 365). Modified Parallel Analysis relies on 
FA. a well understood method which is widely available to users in most statistical pacicages. 
Thus, it is (conceptually and computationally) easier to use than Stout's methods. This study 
will develop a revised and improved version of MPA. 

Parallel and Modified Parallel Analysis 

Parallel Analysis (PA) was proposed by Horn (1965) as an alternative to traditional factor 
analytical methods for identifying the number of latent factors, (e.g. Kaiser, I960. Cattell. 
1966. Bartlett. 1950). 

The rationale behind PA is intuitively compelling, and its application is simple and 
straightforward: Random correlation matrices are generated, and their eigenvalues are 
exuracted and averaged. The eigenvalues of the actual correlations are compared to these 
means and those factors with eigenvalues larger than their counterparts from the 
randomly generated data are retained. Crawford and Koopman (1973). Humphreys and 
Montanelli (1975) and Zwick and Velicer (1986). among others, report that PA works well in 
both Principal Components (PC) and Factor Analysis (FA). Recently Longman. Cota. Holden 
and Fekken ( 1989) published regression equations that eliminate the need to acmally generate 
random matrices for each PA (for the PC case). 

Parallel Analysis is used to determine the true dimensionality of a given dau set, whereas in 
most applications of IRT models one seeks to determine whether a dau set deviates 
significantly fixjm unidimcnsionaUty. Modified ParaUcl Analysis (Drasgow & Lissak. 1983) 
provides an ingenious way of answering this question, using the rationale of PA. Its basic 
stages are: 

(1) The intercorrelations (preferably tetrachoric) of the test's items arc factor analyzed and the 
eigenvalues of the unrotated solution are calculated. 

(2) A "parallel" unidimensional data set is generated by an IRT model. This data set paraUels 
the observed one along all its attributes: It has an equal number of examinees with identical 
abiUtics. and it has the same number of items with identical parameters. Since responses 
are generated by an unidimensional IRT model satisfying the strong local independence 
principle the data set is, by definition, unidimensional. 



ERIC 



6 



-6- 



(3) The (tecrachoric) correlations of the paraUel data set are factor analyzed, aiid the 
eigenvalues of the unrotated solution are calculated. 

(4) The dimensionality of the pool is assessed by comparing the magnitude of the second 
eigenvalues of the two data sets: If the empirical value is "sufficiently close" to the one 
obtained from the parallel data set, the pool is considered unidimensional. 

Drasgow and Lissak (1983) reported five empirical studies providing strong empirical support 
for the procedure. 

Eigenvalue based factor analytical techniques are not always successful in recovering the true 
dimensionality of binary data and, consequently, can't always disunguish between 
unidimensional and multidimensional data sets (e.g. Collins, CUff. McCormick and Zatlin, 
1986; Hattie, 1984; Knol and Bergcr, 1991; Roznowsky, Tucker & Humphreys, 1991; 
Zwick and Velicer, 1986). Thus it may seem surprising that some of the same measures 
perform very well in the framework of PA, and MPA. It is important to stress that the key to 
the success of these methods is their comparative nature. Whatever deficiencies these statistics 
have, they affect equally the results of the two data sets. Both PA and MPA focus on, and 
highlight, whatever differences exist between the empirical and paraUel data sets above and 
beyond the systematic biases that the FA based measures may share. 

Thus, in Hattie's (1984) typology MPA should not be considered a "factor analytic approach". 
In fact, it is closer to the "measures of fit to IRT models". MPA is a general method for 
assessing the similarity, or closeness, between two parallel data sets (one of which is known 
10 be unidimcnsional).Ip this case similarity is quantified by some of the statistics usuaUy 
employed in FA. 

A critique of MPA 

Modified Parallel Analysis suffers from a few technical limitations. In this section we 
describe these limitations and the problems they may cause in applying the method: 

(i) MPA is a randomized procedure, i.e. its results depend to a certain degree on a random 
process, namely, the selection of the parallel data set. Thus, with small enough samples, 
researchers applying exactly the same procedure to the same set of data may reach different 
conclusions because of the variance between the random dau sets generated in their 
simulations. 
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(ii) The simulated and the empirical data sets are equated along most important dimensions and 
any discrepancy between their eigenvalues can ^supposedly, be attributed to the 
muludimensionality of the empirical matrix. Yet, the communaliues are esumated in a 
purely empirical fashion separately for each data set, introducing another important 
difference between them. This factor may bias (in an unknown direction and to an 
unknown degree) the comparative analysis. 

(iii) MPA is a heuristic procedure, i.e. it lacks a measure of sampling variability for the formal 
assessment of the closeness of the criucal statisuc (the second eigenvalue) obtained from 
the unidimensional and the empirical solutions. 



Other important limitadons of MPA are: 

(iv) It compares only the second pair of eigenvalues of the two matrices. This choice lacks a 
solid theoredcal or empirical justificauon, and it may miss differences between the other 
eigenvalues (especially the third). 

(v) N4PA is too limited in its scope. The technique provides a global omnibus test of the 
hypothesis concerning the pool's unidimensionality. It lacks, however, a mechanism to 
follow up rejections of the hypothesized pattern, by eliminating some items and identify a 
unidimensional subset of the pool. 

A REVISED MODEFIED PARALXEL ANALYSIS (RMPA) 

In this section we outline a revised procedure (RMPA) which extends and generalizes the 
MPA. The revised method offers solutions to the technical problems described above 
and incorporates them into the existing framework of MPA. RMPA also includes a second 
stage which allows one to extract unidimensional subsets from larger, potcndally 
multidimensional, pools. ^ 

To solve the first problem we replace tlie random gcneradon of a parallel unidimensional 
population by the theoredcal derivadon of the expected correladons under the assumpuons of 
(1) local independence, (2) unidimensionaUty of the parameter space and (3) the three 
parameter logistic model (e.g. Lord, 1980). The probabiUty of a correct response for item j by 
a test taker with (a single) ability 9i is given by P(Uij = II Si) or, in a shorter notation. P^,. 

p.. -C- + .2) 

""^'^^^ l+exp{-L7aj(er bj)} 
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where a. is the item's disciimination parameter, b. is the item's difficulty and c. is its pseudo- 
guessing probability (see Hambleton, 1983 or Lord, 1980 for details). Under these 
assumptions the expected number of correct answers to any pair of arbitrary items, j and k, in 
a random sample of N examinees is: 



N N 
fj= X Pji and fk= X Pki (3) 
i= I i= I 



Under the assumption of local independence, the expected number of correct answers to both 
items, j and k, is: 

N 



Given f.^, and the two marginals, f. and f^^, the expected 2x^ 'contingency table can be 
constructed, and the expected tetrachoric correlation can be estimated by standard methods 
(e.g. by solving a polynomial using the Newton Raphson method, as suggested by Kendall & 
Smart 1979, pages 324-327). All expectations are (as in the original MP A) conditional upon 
the abilities and item parameters shared by the two data sets. The calculation can be further 
refined when the true distribution of the unidimensional abilities (9i) in the population is 
known. In these cases, the summation is replaced by integration over 9 of the probability 
density function of Qj. 

To solve the second problem wc replace the separate estimation of the coramunalitics in the 
two data sets by the expected tetrachoric correlation between (hypothetical) experimentally 
independent administrations of any item under th^- assumptions of (1) local independence, 
(2) unidimensional ability and (3) a three parameter logistic item curve. This procedure 
amounts to estimating the items' communalities by their expected test-retest reliabilities. It is 
well known (e.g. Lord & Novick, 1968; Mulaik, 1972) that a measure's reliability provides an 
upper bound to its communality. The estimation procedure is just a special case of the 
technique described above for the calculation of the expected correlation. More specifically, if 
we let j=k. Equation 4 is reduced to: 



N 



(4a) 
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The solution of the third problem relies on a data analytic procedure known as "jacknifing" 
(see Arvesen and Salsburg, 1975, Miller, 1974 or Mosteller & Tukey, 1977 for partial 
reviews) \ Assume that the original nxn correlation matrix between the test's items is strictly 
unidimensional. By eliminating one item at a time (i.e. deleting a row, and the corresponding 
column, from the original matrix) we obtain n submatrices of order (n-l)x(n-l) which, by 
definition, are also unidimensional. It is easy to show that under the 'one factor model" (i.e. a 
matrix of rank one), the average first eigenvalue of these n submatrices, scaled by a factor of nJ 
(n-l), is an unbiased estimate of the first eigenvalue of the original intact matrix. 

A useful and important consequence of the "eliminate one item at a time" procedure is that it 
provides a simple method for assessing the impact, or influence^ of any single item on the 
test's eigenvalues. The logic of the MPA procedure predicts that, under unidimensionality, the 
two matrices will have equal eigenvalues. It is generally accepted, and it was confirmed 
empirically by Drasgow & Lissak(1983), that the first eigenvalue (Xy) is approximately equal 
in the observed and the expected matrices, regardless of the dimensionality of the observed 
responses. Thus, except for sampling error, the ratio of the two eigenvalues, RL^, should be: 

RLp X I (observed)/ Xi (expected) = 1 . (5) 

Furthermore, under unidimensionality, the eigenvalues of the n submatrices of the two data 
sets will be similar, will have equal variances and will be highly correlated. Finally, the 
removal of any given item from the pool will affect the observed and the expected dau sets in 
identical fashion and to an equal degree. Thus, equality (5) should also hold in all n 
submatrices obtained by eliminating one item at a time. Let be the first eigenvalue of the 
submatrix obtained after the deletion of item i, and let RL^ be the ratio of the eigenvalues from 
the two parallel data sets. Then, for all items (i=l.*.n), the rafio of the jacknifed 
eigenvalues should equal the ratio of the original values: 

RL^ = X'/ (observed)/xy(expectcd) = RL^ . (6) 

If the responses are unidimensional, similar results are expected to hold for the second, third, 
and all subsequent eigenvalues. If, on the other hand, the observed responses violate 
unidimensionality, the analysis of the two data sets should yield differential results. For 
example, Drasgow and Lissak( 1983) based the original MPA on the prediction that the second 
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eigenvalue of the observed matrix will be larger than its counterpart from the parallel 
unidimensional data set; 

RL,= X,(observed)/ X,(expected) > 1 . (7) 

If the data are generated by a multidimensional model we expect the mean of \he n second 
eigenvalues extracted from the observed submatrices to be larger, and their variance to 
be higher, than their counterparts from the expected data set. Depending on the type and 
degree of deviation from unidimensionality, the correlation between the observed and 
expected values can be low (or even negative). Furthermore, the eigenvalues of the observed 
responses will be more sensitive to the removal of the foreign (or "contaminating") items. 
Since the expected matrix is unidimensional. its eigenvalues should not be affected 
considerably when any arbitraiy item is removed. However, when a contaminating 
item is removed from a multidimensional test, the data set becomes closer to unidimensionality 
and its eigenvalues should decrease. For example, in a test of length n=50 with 8 foreign 
items (8/50= 16% contamination), after the removal of such an item, the level of contamination 
is reduced to (7/49=) 14%. Thus, whenever a contaminating item is eliminated the matching 
eigenvalues should be more similar to each other than in those cases in which a regular 
(noncontaminating) item is removed. Consequently, the ratio of the eigenvalues should be 
closer to unity in these instances. 

To summarize, for any given data set, the ratio between the first eigenvalues, RL,, in the two 
data sets can be used as a b 3nchmark against which one can assess and test the ratios derived 
from the second and third eigenvalues (RL2 and RL3, respectively). At the global (i.e. test or 
pool) level, this approach is attractive because the behavior of RLj and RL3 is assessed by a 
data based index which is more sensitive to, and reflects, the pecuUarities and idiosyncrasies of 
the specific test being examined. At the local (i.e. item) level, this procedure provides a natural 
way of ranking, and scaling, the items according to their deviation from the pattern expected 
under unidimensionaUty. These properties can be used to develop a procedure for testing the 
global dimensionahty of the observed responses, and a method of selecting uuidimensional 
pools. In the next section we describe tiie technical detaUs of such a testing procedure. 

The "gap tes: ' 

To faciUtate the comparison of the two data sets we calculate, for aU items (i=l...n) and for tiie 
first k eigenvalues (k=l,2,3), the ratio of the two matched eigenvalues: 



RL"^' = X;!(observed)/ >^'(expected) 

II 



(8) 
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The global ratio RL^ as well as the individual RL (i=l..n), are insensitive to the 
dimensionality of the observed data set. Their empirical distribution will be used to test the 
hypochesis that the ratios of the second and third eigenvalues behave similarly. Formally, we 
wish to test that F{RL;^} = F{I^;}, and F{RL3'} = FfRL'/K where F{'} stands for the 
distribution of the relevant statistic. The alternative hypothesis is that the ratios are distributed 
differentially. 

We are particularly interested in the case where an essentially unidimensional data set is 
contaminated by a second (sometimes called "nuisance") ability. We speculated earlier, tliat 
removal of such contaminating items will affect differentially the two matched eigenvalues. 
When analyzing the correlations from the observed responses we expect to observe two 
distinct clusters of eigenvalues from the unidimensional and the contaminating pool, 
respectively — separated by a substantial "gap". No parallel clustering and separation is 
expected in the conesponding eigenvalues of the mauix of expected correlations. 

To detect such unusual gaps we adopt a procedure described by Wainer and Schacht (1978) to 
detect unusually large gaps in smngs of ordered values. The first step in this procedure is to 
rank order the values in descending order and to calculate the (n-1) gaps, g., by subtracting 
each observation from the immediately previous (i.e. larger) one. The gaps are then weighted 
by a set of logistic weights to yield weighted gaps» y.. These weights were selected to account 
and compensate for the fact that, typically, observations are more dense (hence should be 
overweighted) near the center and more sparse (and should be undcrweighted) in the tails of 
the distribution. Formally: 

y\ = Vi(n.i)gT . (9) 

Finally, these values arc standaiujized by division by y^^, the midmean (i.e. the mean of the 
central 50% values) of the weighted gaps. Thus, me standardized weighted gaps (SWGs 
for short), z. can be expressed as: 

Zero gaps indicate that two adjacent observations arc equal, and unit gaps indicate .at the 
distance between two observations is equal to the gaps' midmean. By definition, all gaps are 
non-negative but are unbounded from above. Wainer and Schacht (1978) suggest that z. 
values greater than 2,25 indicate "unusually" large gaps. The probability of observing gaps 
this wide by chance is approximately 0.03 under the nomnal distribution, but this value was 
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shown by Wainer and Schacht (1978) to work quite well for a variety of symmetric t 
distributions with tails larger than the nonnal. 

We will use this procedure to detect the location of the gap separating the items from the two 
pools, on the basis of ratios of the matched eigenvalues, RLj^' (k > 1), Thus, the hypothesis 
will be tested by comparing MAX(Zj^), the largest SWG, with a critical rejection threshold. 
However, in the absence of precise information regarding the form of the distribution of these 
ratios, and the multiplicity of tests involved, it is not sufficient to rely on the 2,25 universal 
mle of thumb proposed by Wainer and Schacht. Instead, we fmd it necessary to develop more 
conservative rejection rules . 

There are various ways of deriving critical rejection points for this decision: one can estimate 
the desired percentiles (.01, .05, etc.) from the distribution of RL/, or use a version of 
Chebyshev inequality (e.g. Stuart and Ord, 1987. page 1 10). The regular Chebyshev 
inequality states that the probability of fmding a value located more than K standard deviations 
(SDs) from the population's mean is smaller than 1/K^ for any distribution with finite 
moments; A tighter version, invoking the additional assumptions that the disuibution is 
symmetric and unimodal, yields a lower upper limit (4/9K^), for the probability of the same 
event ^ 

The decision, to reject Hq, will be based on a comparison with a critical threshold, T(z^). The 
threshold is derived from the distribution of the ratios of the first eigenvalue, in the same 
data set. Specifically, for k=2,3 we will reject Hq if: 

MAX(Z^^)>T(Zj)=:(Mj-hKSj) 

where and arc the mean and SD. respectively, of the SWGs, z^^, calculated from the 
ratios of first set of matched eigenvalues, RL 

If MAX{z^^ and/or MAX(23.) > T(Zj), i.e. there is a significant gap in either distribution of 
ratios, we can eliminate those items which are located above the significant gap(s)'*. Let m^ 
denote the number of items eliminated (m^ > 0) after this first pass through the data. Repeat 
the whole process with the reduced (n-m^)x(n-ra^) correlation niatrices. This second analysis 
may lead to the ehniination of additional (say m^) items. Repeat the procedure with the 
remaining items, and stop when the test fails to detect items to be rejected. 
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AN EMPIRICAL STUDY OF RMPA 
Method 

In this section we report results of an empirical study designed to test RMPA. Like most other 
studies in this area we simulated artificial test results by combining real item parameters and a 
set of reasonable assumptions regarding the distribution of abilities in the population of test 
takers. For the purpose of this study we contaminated a large unidimensional pool by (various 
proportions of) responses generated by a second (nuisance) ability correlated (at various 
levels) with the Rrst. The efficiency of the RMPA was assessed by its ability to identify 
correctly the contaminating items and, consequently, partition the test into its two basic 
components. 

We expect this procedure to be most efficient in cases of approximate unidimensionality. In 
other words, it should-detect accurately relatively low levels of contamination, but not mixtures 
of two (equal) abilities. We also predict that the accuracy of the detection will be inversely 
related to the correlation between the two abilities involved. 

Design 

We generated 20 distinct "artificial tests". The following characteristics were fixed for all the 
tests: 

n = test length = 80 items; 

N = sample size = 2000 examinees; 

t = number of abilities = 2. 

The following variables were manipulated across tests: 

p = proportion of contaminating items = 0%, 10%, 25% or 50% (p=0% is a a stricUy, 
uncontamdnatcd, unidimensional test and the other three cases represent low, medium 
and high levels of contamination); 

r = the correlation between 9^ and 9^, the two abUities = 0.0, 0.5, 0.7 (the three values are 
approximately equally spaced in terms of r^). 

Replications: All combinations of p and r were replicated twice (i.e. with different seeds for 
the generation of the abiUties, and different item parameters). In the sequel the two replicaUon,^ 
O arc labeled "B" and "R'\ 
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With the exception of the control condition (p=0, r=0), this can be viewed as a 3 x 3 factorial 
design repeated twice. 

Item Parameters 

The items for half the tests (replication "R") were randomly selected from the item bank of a 
test of English as a Foreign Language (EFL). This test was developed and is routinely used 
by the National Institute for Testing and Evaluation (NTTE) as part of the Psychometric 
Entrance Test (PET) which is administered to all applicants to universities in Israel. The item 
parameters were estimated under the three parameter logistic model (Equation 2) using 
responses from approximately 7,000 examinees who took the test in 1988. The estimation 
was performed using the NTTEST parameter estimation program (Cohen & Bodner, 1989). 
These parameter estimates for the n=80 items will henceforth be referred to as "true 
parameters". 

The items for the other 10 tests (repUcation "B") were generated artificially, according to some 
distributional assumptions: The discrimination parameters (a's) were sampled from a normal 
distribution with a mean of 1.1 and a s.d. of 0.3; The difficulty parameters (b's) were 
obtained from a normal distribution with a mean of 0 and a s.d. of 0.8; The pseudo-guessing 
parameters (c's) are taken from a uniform distribution over the range 0. 1 - 0.3. The values of 
the three parameters were sampled, from the respective sources, independently. 

Table 1 summarizes the information regarding the two sets of true parameters. The two tests 
are equally difficult, but vary with respect to other aspects. The discrimination parameters of 
the real items ("R") have a higher mean and variance (m^=1.33 and s^=0.5 1) than the artificial 
ones ("B") (m =1.12 and s =0.25). On the average, it is easier to guess in the artificial test 
(m =0.2 vs. 0.16). Finally, whereas the parameters of the artificial items are uncorrelated (by 

c * 

design), the values of the EFL items parameters arc moderately correlated. 



Insert Table 1 about here 
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Abilities 

All samples include N=2000 simulated "respondents". First we generated four mutually 
uncoirelated sets of abilities (T, A^. A^ and A3): We sampled 8000 independent observations 
from the standard (0.1) normal distribution and randomly assigned them to the four sets. 
Correlated abilities were generated by calculating: 

T(r) = r-T+ V 1 - • Ai (^^ 



where A. stand for A^, A^ or A3, and r is the desired correlation (0.0, 0.5, 0.7) between the 
new set of abilities. T(r). and the reference set. T. Thus T(0), T(.5), T(.7) are sets of N=2000 
normally distributed abilities which correlate 0.0. 0.5 and 0.7, respectively, with T. 



Responses 

Four sets of unidimensional response vectors were generated. Each set was simulated with a 
different set of abilities {T, T(0). T(.5) or T(.7) }. and aU responses were generated with the 
"true" item parameters. The response vectors were simulated with the NTTECAT software 
package (Cohen. Bodner & Ronen, 1989), which implements the process described by 
Drasgow and Lissak (1983). 

The vectors generated with the T abilities are considered the "original" responses based on the 
dominant ability. Contaminated responses were obtained by replacing the original responses 
on p% of the items (randomly selected) with the corresponding responses generated by one of 
the other samples of abilities. Note that for the case of n=0 this procedure simulates a two- 
dimensional "noncompensatory" model (e.g. Ackerman, 1989, Sympson, 1978), whereas the 
other cases (r > 0) simulate "compensatory" models (e.g. Ackerman, 1989, Reckasc, 1985). 



In each of the artificial tests the three parameters of the n=SO items were estimated with the 
NTTEST program (Cohen & Bodner, 1989). These are the various sets of "estimated 
parameters", to be used in the generation of the expected correlations. 

Consistent with the massive Uterature on this topic (e.g. Dorans & Kingston, 1985; MUler 8l 
Oshima, 1992; Oshima & Miller, 1992), we found that the estimates of the b's and c's were 
not affected by the contaminauon. However, the estimates of the a's (the discrimination 
parameters) are sensitive to the level of contamination. Consistent with other studies in the 
literature, the estimates for items loaded on the dominant abiUty are hardly affected, whereas 
O the discrimination measures of the contaminating items are reduced considerably. The 
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magmtude of this "shrinkage" is ^elated to the level of contamination and the correlation 
between the two factors. 

Results 
Standard MPA 

The standard MPA procedure, prescribed by Drasgow and Lissak (1983), was performed. 
Table 2 summarizes the results of this analysis. The table displays the first 
three eigenvalues of both correlation matrices, as well as their ratios. 



Insert Table 2 about here 



There is a clear and consistent pattern in the data which can be summarized by three 
observations: 

(i) The first eigenvalues are, practically, equal in the two matrices and their ratio is, 
essentially, 1. There are no discernible differences between the 18 contaminated data sets 
and, in this respect, they are indistinguishable from the two uncontaminated tests. 

(ii) In all contaminated tests, the second eigenvalue of the observed matrix is larger than its 
expected counterpart. Conscquendy, their ratio is greater than unity, as predicted by 
Drasgow & Lissak (1983). The ratio is a monotonically increasing function of p, the level 
of contaminadon, and a monotonically decreasing fiincLon of r, the inter-ability 
correlation. 

(iii) The ratio of the third pair of eigenvalues is also greater than one. In fact, in most cases it 
is greater than the second ratio. The third rado is not systemadcally related to r, the inter- 
ability correlation. However, it increases monotonically as a function of p, the level of 
contamination. The sharpest effect is obtained for highly (r=0.7) correlated, and the 
weakest effect is found for uncorrelated (r=0.0) abilities. 

EMEA 

We performed an informal RMPA by examining the eigenvalues of the jacknifed parallel 
matrices. Table 3 displays means, and standard deviations, of the first three eigenvalues 
extracted from the jacknifed submatrices. All the values in the table arc based on n=80 
matrices of order (n-l)=79. Note that the mean values are related to the eigenvalues from table 
2 through multiplication by a scale factor of n/(n-l)=80r79, 
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Insert Table 3 about here 



Table 4 presents ratios of the means, and the variances, of the three jacknifed eigenvalues of 
the 20 tests. 



Insert Table 4 about here 



There is a close correspondence between these mean ratios and the ratios presented in table 2, 
and the same three basic conclusions apply here, as well. The ratios of the variances follow a 
similar, but not identical, pattern: 

(i) The variances of the first eigenvalues are, on the average, very close to each other and their 
ratio is close to unity. The only exceptions are the cases {r=0, p=50}, which represent 
mixtures of two unidimcnsional half-tests involving uncorrclated abilities. 

(ii) In most cases (and on the average) the variance of the second (jacknifed) eigenvalues in the 
observed matrices is higher than in the expected one. The effect is most pronounced in the 
case of the independent traits (r^), and for moderate or high levels of contamination 
(p=25 and 50, respectively). 

(iii) In all 20 tests the variances of the third Gacknifcd) eigenvalues arc substantially higher m 
the observed matrices. The effect is much stronger than for the second eigenvalue, but 
there is no systematic pattern of change across levels and types of contamination. 

Table 5 presents the correlations between the matched jacknifed eigenvalues for the 20 tests. 
Each correlation is based on n=80 observations. 



Insert Table 5 about here 
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The pattern of results is clear and consistent with our expectations: 

(i) There is a high (almost perfect) linear correlation for the first eigenvalue in most tests. The 
single exception is the {Rep=R, r=0, p=50} case, which is a mixture of two uncorreiated 
(unidimensional) half-tests. 

(ii) In all cases of moderate and high contamination (p=25 and 50, respectively) the 
correlations based on the second and third eigenvalue are low, or negative. 

(iii) In most cases of low contamination (p=10) the correlations based on the second eigenvalue 
are high (almost like for the first eigenvalue), but the correlations based on the third 
eigenvalue are always low, or negative. 

This pattern indicates that, as suggested by Drasgow and Lissak (1983) and others, the first 
eigenvalues of the two parallel matrices are practically indistinguishable, across all types and 
levels of contamination. However, contrary to Drasgow and Lissak's speculation, not all the 
differences between the two data sets can be detected by comparing the second pair of 
eigenvalues. The means, variances and correlations of the jacknifed values seem to suggest 
that in some cases of low contamination (p=0. 10) departure from unidimensionality can only 
be detected by examining the third pair of eigenvalues. 

Rejection Thresholds 

Tablf; 6 presents six rejection thresholds calculated from the distribution of the first ratio in the 
20 tests. These six rejection thresholds are obtained by crossing Dvo conrldence levels (95% 
and 99%) with three rules of detection ~ an empirical value, a value calculated by the "tight" 
(i.e. assuming unidimodality and symmetry) Chebyshcv inequality, and a value derived from 
the unconstrained Chcbyshev inequality. 



Insert Table 6 about here 



In all tests, and for both confidence levels, the empirical percentile is more liberal than the 
corresponding Chebyshev bounds. Thus* the three rules can be ranked, from the most to the 
least conservative, identically for all tests and for both levels of confidence: 

Unconstrained Chebyshev > Constrained Chebyshcv > Empirical 
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One remarkable and reassuring aspect of this table is the relatively low variance of the bounds 
across the various conditions and replications. This indicates that the ratio of the first pair of 
jacknifed eigenvalues has a relatively stable distribution across the levels and types of 
contamination. 

The most important issue, from a practical point of view, is to choose the "best" threshold for 
detection of wide gaps. To address this issue we focus on the performance of the various 
indices in the uncontaminated (p=0) case. Table 7 displays tlie proportion of SWGs exceeding 
the various indices for the three ratios. Since this is a strictly unidimensional test, we expect 
this proportion to be invariant for all three ratios and not to exceed its nominal confidence level 
(95% or 99%). Clearly, the empirical percentiles fail the invariance requirement and the 95% 
constrained Chebyshev bound is too liberal for the third ratio. In light of these results we 
conclude that is best to identify as "unusually wide gaps" those values that exceed the 95% 
unconstrained, or the constrained 99% Chebyshev bounds. We will focus primarily on 
rejections with 99% confidence. However, for completeness sake, we will report in the sequel 
results according to all the thresholds. 



Insert Table 7 about here 



Partition of the Tests 

Tables 8a - Sclist the maximal SWGs observed in the distributions of the three ratios for each 
test. The tables also display the pattern of significance achieved by this maximal SWG, and its 
location. The columns labeled "significance" simply count how many ^'of the increasingly 
stringent) thresholds were exceeded in each family of tests. In the 95% and 99% columns, a 1 
indicates that the observed value is greater than the empirical percentile lower than both 
Chebyshev bounds; a value of 2 describes a situation where the actual value is greater than the 
constrained (but smaller than the unconstrained) Chebyshev bound, and a value of 3 denotes a 
case where the maximal gap is larger than the most severe rejection rule. Our previous results 
(sec table 7) dictate to interpret as "significant" values of 2 (at 99%), or values of 3 (at 95%), 

The location of the gap is described by reporting the number of items above, and below, it. 
Recall that according to the logic of RMPA the contaminating items should have lower (i.e. 
closer to unity) ratios. We rank ordered the rados in ascending order, so these items are 
expected to cluster "above" the gap. As a rule, we expect the proportion of item above the gap 
Q to match, approximately, the proportion of contamination in the specific test. 

EBJC 20 



-20- 



We reject the null hypothesis of unidimensionality if: 

(1) The number of items "above the gap" < nil AND 

(2) The Maximal S WG of the second AND/OR the third ratio is greater than the designated 
rejection threshold. 

We examine three rejection rules with decreasing levels of conservatism: (1) 99% according to 
an unconstrained Chebyshev inequaiiry, (2) 99% according to a constrained Chebyshev 
inequality, and (3) 95% according to the unconstrained Chebyshev inequality. 



Insert Tables 8a - 8c about here 



As expected, there are no significant gaps in the distribution of the first ratio but, in most tests, 
the largest S WG in the distribution of the second and/or third ratio is significant. We examine 
these significant gaps according to the three valid rejection thresholds: 

All six cases with low (p=10) contamination are significant at the 99% level (five of them by 
the most severe criterion). In all six cases the gap separates the top 10% items from the bottom 
90%. It appears that the procedure works well for this type of contamination. 

Only three of the highly contaminated tests (p=50) are significant at 99%. More important, 
however, is the fact that in all six tests the widest gap is located at the bottom of the 
distribution. Although the numbers vary sUghtly across tests, the proportion of items above 
the gap is always greater than 80%. Clearly, the gap test docs not work well for a mixture of 
two half tests. 



The pattern of results is sUghtly more complex in the case of moderate (p=25) contamination, 
and it depends on the level of the inter-ability correlation: For both tests with unconelated 
(r=0) abilities, and one of the tests with moderately correlated (r=0.5) abilities, the significant 
gap (99%) in the distribution of the second ratio separates the upper 25% items from the rest of 
the test. In the other test with r=0.5 the gap between the top 25% of the items and the lower 
75% is significant at the 95% level. Finally, for the tests involving highly correlated abiUties 
(r=0.7), the maximal gap is located at the lower end of the distribution (69 and 72 items above 
the gap). In both cases the second largest gap distinguishes between the (most) contaminating 
items and the original ones. Thus, the gap test operates well only for cases with low inter- 



n^i^" ability correlations. 
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To summarize, ElMPA found a significant gap in the distribution of the ratios of matched 
eigenvalues in all the tests examined. In 14 tests the gap was significant at 99% and in the 
other six at 95%. A significant gap located in the upper half of the distribution (i.e, with fewer 
items above the gap than bellow it) is taken as a strong indication of violation of 
unidimensionality and prescribes elimination of all items above the gap. The ten tests 
identified by this criterion include all those with low contamination (p=10), as well as the 
moderately contaminated ones (p=25), with moderate level of inter-ability correlation (r <0.7). 

In the sequel we focus only on these 10 shortened tests. Plots of the 10 relevant distributions 
of standardized weighted gaps (not presented here because of space limitations), clearly show 
that: 

(i) the contaminating items are clustered at one end of the distribution, and 

(ii) there is an unusually large gap separating this cluster from the bulk of the items. This gap 
can be detected in the raw gaps, but it is more pronounced in the standardized weighted 
form. 

The quality of the technique is assessed by its ability to detect the contaminating items and 
remove them, while retaining the original ones. Table 9 summarizes this analysis for the 10 
short tests. For each one we report the hit rate (i.e. contaminating items rejected correctly) 
and the false alarm rate (i.e. original items rejected incorrectly). The figures are very 
impressive — for all the tests with p=10%, the hit rate is 100% and for the tests with p=25% 
it is 95%. Both figures are accompanied by false alam rates close to 0. 



Insert Table 9 about here 



Re-cxamination of the s hortened tests 

Having shortened 10 tests according to the results of the initial RMPA we repeated the 
procedure. The second iteration verifies the unidimensionality of the shortened tests: If the 
first stage is successful in removing all sources of contamination, we do not expect to detect 
any significant gaps in this second round. 

Tables 10 and 1 1 report the results of the MP A and the RMPA of the shortened tests. A quick 
comparison with tables 2 and 4 (summarizing the same results for the original fiill tests) 
reveals that ail major sources of multidimcnsionality were eliniinated. The ratios of the second 
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eigenvalues, and the ratios of their variances, are close to unity (We assume that a heuristic 
MPA would also declare all these tests unidimensionai). The third ratios are somewhat higher 
but are, considerably, lower than those of the original tests. 



Insert Tables 10 and 1 1 about here 



The SWGs of the remaining items were calculated, new rejection thresholds were derived, and 
the gap test was applied again ^ 

The only significant gap was found in the {Rep=B r=0.7 p=10} test. In this case the second 
iteration of the RMPA prescribes removal of five additional items. All contaminated items 
were successfully detected by the first iteration so these are five "false alarms". The final test 
consists of 66 unidimensionai items (instead of 72). 
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SUMMARY 

The goal of the current research was to develop a practical, yet theoretically sound and 
computationally feasible, tool for testing the global dimensionaHty of large item pools and 
eliminating items which cause violations of the pool's unidimensionality. Both goals are 
attained in the unified framework of revised modified parallel analysis (RMPA). 

RMPA is an extension and generalization of the modified parallel analysis (MP A) method that 
was developed by Drasgow and Lissak ( 1983) as an approximate method for testing the 
unidimensionality of item pools. MPA replies on a heuristic comparison of a statistic (the 
second eigenvalue) derived from the matrix of items' intercorrelations and the corresponding 
value extracted from a "paraUel" matrix generated by a unidimensional, and locally 
independent, model (in our case the three parameter logisdc model). 

RMPA is based on a similar comparative logic, but improves upon MPA in several ways: 

(1) It alleviates some minor technical limitations, through the use of expected inter-item 
correladons and item communalides. WhUe in MPA the inter-item correlations are 
generated by a randomized procedure, in RMPA their expected values are derived 
theoretically under the exact assumptions of the 3-P logistic model. 

(2) It implements a formal test for comparing the observed data set with its parallel (and 
unidimensional) counterpart. The test is based on inspection of the ratio of the k'th 
eigenvalue of the observed intercorrelation matrix to that of the expected matrix. The ratios 
for the second and third eigenvalues are compared with the ratio of the first eigenvalue 
which serves as a benchmark. In case of a unidimensional data set, all three ratios should 
be of equal magnitude. Violations of unidimensionality are manifested in larger observed/ 
expected ratios of the second and third eigenvalues relative to that of the first 

(3) Contingent upon the results of this test, it provides a method for identifying and 
eliminating items which violate the test's unidimensionality. 

The testing and elimination procedures are based on tiie "eliminate one item at a time" 
principle. This methodology allows one to assess tiie contribution of each item to tiie test's 
eigenvalues. Furtiiermore, one can determine tiie variance and distribution of these values and 
analyze tiie differential impact of any given item in tiie observed and parallel matrices. Items 
which have a "significantiy" larger impact in tiie observed data set violate unidimensionality. 
The detection of tiiese items relies on a conservative version of Wainer & Schacht's (1978) 
Q "gapping" test. 
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The lagest (first) eigenvalues of the observed ai.d expected matrices are practically identical in 
all cases, regardless of the level of correlation between the two abilities and the degree of 
contamination. Therefore, we used the distribution of their ratio to determine rejection 
thresholds for the ratio of the second and third eigenvalues. These thresholds are based on 
conservative Chebyshev bounds, and are specifically tailored to each data set. 

RMPA was tested in several simulations of unidiiuensional item pools which were 
contaminated by various proportions of items loaded on a secondary nuisance ability. The 
method was highly successful in identifying low (10%) levels of departure from 
unidimensionality, and in detecting moderate (25%) deviations from unidimensionality when 
the abilities were not highly (r < 0.7) correlated. In these cases over 90% of the contaminating 
items were identified and less than 1% of the original items were eliminated en-oneously. The 
procedure failed however, in tests which are easily detected as multidimensional by the MPA 
method, namely tests which are equal mixtures (50%) of items loaded on two abilities. 

The two methods, MPA and RMPA, can be regarded as complementing each other. Clear 
cases of multidimensionality can be easily detected by inspecting the observed/expected ratio of 
the second eigenvalue but the gapping test is unable to separate between the two 
(unidimensional) sub-pools. Paradoxically, in cases of multidimensionality which are not 
easUy detected by the magnitude of the ratio, the gapping test not only detects the presence of 
nuisance ability factor but also identifies the contaminating items. The fmdings of this study 
suggest that practical application of RMPA should proceed in two stages. In the first stage, the 
ratio of the second eigenvalues (observed/expected) should be inspected. In the second stage 
of the analysis the gapping test is appUed. The decision as to the unidimensionaUty of the data 
set is determined by a conjunction of the results from both stages. Therefore, the 
unidimensionaUty hypothesis should be rejected if the results of either stage indicate 
multidimensionaUty. Large ratio (in the first stage) combined with nonsignificant gaps (in the 
second stage) most probably indicate a case of a multidimensional data set with close to equal 
proportions of items loaded on each ability factor. 

It remains to be proved whether RMPA is as effective in separating out items that load on more 
than two factors and, more generally, whether the procedure is appUcable to other factor 
structures. 

We conclude by pointing out that the logic of MPA and RMPA can be generalized to other 
suusucs of closeness between the two data sets. For example, it might be interesting to apply 
O it to indices derived from non linear factor analysis (e.s. McDonald, 1982). 
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FOOTNOTES 



( 1 ) Strictly speaking "jacknifing" refers to an analysis in which observations (i.e. respondents) 
are eliminated one at a time from the sample. In this case, we eliminate variables (items) in 
a similar fashion. Several item analysis computer programs use a similar approach in order 
to identify subscales with maximal reliability. 

(2) I., the sample influence function (Devlin, Gnanadesikan and Kettenring, 1975; Hampel, 
1974) of parameter, T, is given by: 

I. = (n-1)(T-T;), 

where n is the number of items, and T . is an estimate of the parameter T obtained after the 
elimination of item i. Note that I is, simply, a linear transformation of T 

(3) StricUy speaking. Chebyshev inequality requires knowledge of the parameters (mean and 
variance) of the population of interest. However, Saw, Yang and Mo ( 1984) have shown 
that sample estimates of these parameters can be used, with very little loss of precision, in 
moderately large samples. 

(4) Occasionally a large (and significant) gap will be detected in the lower tail of the 
distribution, i.e. separating the bulk of the data from a minority of items with unusual low 
ratios of observed/expected jacknifed eigenvalues. Clearly, these cases are not relevant for 
our hypothesis. 

(5) Since the procedure is data driven, we opt not to use the thresholds values employed in the 
first stage. Thus, when analyzing a test consisting of (n - mj) items one should obtain the 
same results, and reach the same conclusions, whether it is treated as "an original" or "a 
reduced" test. 
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Table 1: 

Means, standard deviations and correlations of the two sets of item parameters 



1 r 

Rep=B 

Parameter n Mean Std. Dev. 




Rep=R 

Parameter n Mean Std. Dev. 


a 80 1.123 0.245 
b 80 0.172 0.873 
c 80 0.202 0.057 




a 80 1.328 0.511 
b 80 -0.026 0.985 
c 80 0.161 0.098 


Correlations 
a b c 




Correlations 
a b c 


a 1.000 0.090 -0.196 
b 0.090 1.000 -0.260 
c -0.196 -0.260 1.000 




a 1.000 0.518 0.397 
b 0.518 1.000 0.754 
c 0.397 0.754 1.000 



31 



-31 - 



Table 2: 

Modified Parallel Analysis {MP A) of 20 tests: 
The first three eigenvalues for the observed and 
expected matrices, and their ratios 



Rep 



F.i gen value 1 



F.i gen value 2 



Eigenvalue 3 



Exp Obs Obs/Exp Exp Obs Obs/Exp Exp Obs Obs/Exp 







n 


24. 34 


25.10 


1.03 


D 


U.U 




97 03 


22.71 


1.03 


r> 


U.U 


LJ 


18 15 


18.80 


1.04 




0 0 


50 


11.93 


12.58 


1.05 


B 


0.5 


10 


22.72 


23.55 


1.04 


B 


0.5 


25 


19.93 


20.77 


1.04 


B 


0.5 


50 


17.76 


18.71 


1.05 


B 


0.7 


10 


23.31 


24.16 


1.04 


B 


0.7 


25 


21.27 


22.13 


1.04 


B 


0.7 


50 


19.91 


20.82 


1.05 


R 




0 


26.20 


26.23 


1.00 


R 


0.0 


10 


23.59 


23.84 


1.01 


R 


0.0 


25 


19.57 


19.81 


1.01 


R 


0.0 


50 


12.23 


12.90 


1.05 


R 


0.5 


10 


23.68 


24.14 


1.02 


R 


0.5 


25 


21.45 


21.90 


1.02 


R 


0.5 


50 


19.30 


19.88 


1.03 


R 


0.7 


10 


24.45 


24.86 


1.02 


R 


0.7 


25 


22.91 


23.30 


1.02 


R 


0.7 


50 


21.78 


22.25 


1.02 



1.79 


1.78 


0.99 


0.17 


0.67 


1.62 


2.55 


1.58 


0.17 


1.66 


1.43 


5.59 


3.92 


0.15 


1.53 


0.61 


12.01 


19.77 


0.08 


0.85 


1.66 


1.78 


1.07 


0.18 


1.66 


1.46 


3.87 


2.65 


0.16 


1.54 


0.92 


6.05 


6.60 


0.07 


1.07 


1.66 


1.73 


1.04 


0.18 


1.26 


1.46 


2.30 


1.58 


0.15 


1.55 


1.18 


3.67 


3.11 


0.09 


1.34 


3.47 


3.22 


0.93 


0.36 


0.63 


2.84 


2.86 


1.01 


0.25 


2.57 


2.38 


6.20 


2.60 


0.20 


2.33 


1.45 


12.20 


8.39 


0.15 


1.77 


2.76 


2.76 


1.00 


0.28 


1.90 


2.48 


4.31 


1.74 


0.25 


2.45 


1.79 


6.35 


3.55 


0.13 


1.93 


2.88 


2.87 


1.00 


0.30 


1.18 


2.64 


3.16 


1.20 


0.23 


2.28 


2.32 


3.89 


1.68 


0.17 


2.39 



4.02 
9.72 
10.17 
10.35 
9.13 
9.75 
14.38 
7.09 
10.51 
14.26 

1.78 
10.20 
11.91 
11.53 

6.69 

9.70 
14.89 

4.01 
10.09 
14.17 



Notes: 

All results based on n=80 items and N=2000 respondents. 
Exp = Derived torn matrix of expected correlations 
Obs = Derived from matrix of observed correlations. 
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Table 3: 

Revised Modified Parallel Analysis (RM PA) of 20^ tests: r,u - i ^ 
Means and standard deviations of the first three eigenvalues of the jackmfed 
submatrices (observed and expected] 



Rep r 



Source 



]Rigftn value 1^ 
Mean 



SD 



Ri gen value 2 
Mean SD 



Eigenvalue 3 

Mean SD 



B . 0 Exp 24.036 

Obs 24.786 

B 0.0 10 Exp 21.752 

Obs 22.429 

B 0.0 25 Exp 17.920 

Obs 18.563 

B 0.0 50 Exp 11.778 

Obs 12.432 

B 0.5 10 Exp 22.437 

Obs 23.255 

B 0.5 25 Exp 19.685 

Obs 20.507 

B 0.5 50 Exp 17.536 

Obs 18.472 

B 0.7 10 Exp 23.023 

Obs 23.858 

B 0.7 25 Exp 21.008 

Obs 21.850 

B 0.7 50 Exp 19.660 

Obs 20.564 

R - 0 Exp 25.874 

Obs 25.899 

R 0.0 10 Exp 23.292 

Obs 23.547 



R 0.0 25 Exp 19.330 

Obs 19.561 

R 0.0 50 Exp 12.078 

Obs 12.735 

R 0.5 10 Exp 23.389 

Obs 23.835 

R 0.5 25 Exp 21.181 

Obs 21.629 

R 0.5 50 Exp 19.063 

Obs 19.636 

R 0.7 10 Exp 24.145 

Obs 24.552 

R 0.7 25 Exp 22.627 

Obs 23.006 

R 0.7 50 Exp 21.504 

Obs 21.970 



0.122 


1.764 


0.029 


0.165 


0.004 


0.1 18 


1.756 


U.U27 


n AAA 
U.0O4 




0.145 


1.598 


0.026 


0.168 


0.004 


0.147 


2.515 


r\ 1 r\i 
U. IUj 






0.163 


1.409 


0.027 


0.148 


0.004 


0.169 


c c 1 n 


U. 143 


1 . JU / 




0.136 


0.600 


0.013 


0.081 


0.002 


0.158 


1 1 .842 


U.IDj 




yj.yj i\j 


0.133 


1.640 


0.027 


0.179 


0.004 


0.128 


1.707 


U.UjO 


L .U JVJ 


0.038 


0.136 


1.443 


0.026 


0.156 


0.004 


0.128 


3.0 lo 


U.U{54 


i ,J LJ 


0.027 


0.089 


0.906 


0.014 


0.073 


0.002 


0.085 




U.Uj J 


1 n^*^ 

L .UJ J 


0.019 


0.125 


1.640 


0.026 


0.175 


0.004 


0.119 


1.713 


U.UZj 


1 0/11 




0.126 


1.437 


0.024 


0.145 


0.003 


0.122 


2.267 




1 ^ofi 

i . JZO 


0.027 


0.103 


1.164 


0.019 


0.093 


0.002 


0.101 


3.625 


U.UZj 




n 09 1 


0.128 


3.424 


0.040 


0.351 


0.006 


0.132 


3.178 


U.UJO 


u.ozo 


n 007 

VJ .\J\J 1 


0.155 


2.808 


0.040 


0.249 


0.006 


0.16O 


2.829 


U.UJj 


Z.J J i 


0 1 1 


0.174 


2.354 


0.039 


0.193 


0.005 


0.182 


£ 111 

0.121 


u. i4y 




0 01*5 


0.142 


1.436 


0.031 


0.152 


0.0O4 


0.187 


12.045 


U. 1 / J 


! 7f 1 


0.037 


0.141 


2.726 


0.038 


0.281 


0.006 


0.138 


2.720 


U.UJ*V 


1 9nf\ 

L .O / U 


0.078 


0.152 


2.449 


0.038 


0.249 


0.006 


0.143 


4.251 


0.084 


2.415 


0.035 


0.107 


1.763 


0.023 


0.128 


0.004 


O.lOl 


6.263 


0.037 


1.906 


0.025 


0.136 


2.839 


0.038 


0.291 


0.006 


0.132 


2.8j5 


0.033 


1.166 


0.043 


0.138 


2.605 


0.037 


0.223 


0.005 


0.132 


3.123 


0.040 


2.252 


0.033 


0.118 


2.292 


0.030 


0.166 


0.004 


0.116 


3.843 


0.027 


2.357 


0.030 



ERIC 



Notes: 



All results based on n=80 items and N=2000 respondents. 
Exp =5 Derived firom matrix of expected correlations 
Obs =5 Derived from matrix of observed correlations. 
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Table 4: 



Revised Modified Parallel Analysis (RMPA) of 20 tests: 

Ratio of means and variances of eigenvalues of the jacknifed submatrices 

(Ratio = observed / expected) 



Rep 



B 




0 


B 


0.0 


10 


B 


0.0 


25 


B 


0.0 


50 


B 


0.5 


10 


B 


0.5 


25 


B 


0.5 


50 


B 


0.7 


10 


B 


0.7 


25 


B 


0.7 


50 



R 




0 


R 


0.0 


10 


R 


0.0 


25 


R 


0.0 


50 


R 


0.5 


10 


R 


0.5 


25 


R 


0.5 


50 


R 


0.7 


10 


R 


0.7 


25 


R 


0.7 


50 



_ lvalue I Eigenvalue 2 I;ig?pva]ug 3 

Mean Var Mean Var Mean Var ■ 



1.031 


0.933 


0.995 


0.884 


4.030 


6.007 


1.031 


1.034 


1.574 


15.449 


9.730 


44.102 


1.036 


1.074 


3.917 


28.726 


10.187 


52.047 


1.056 


1.355 


19.748 


151.159 


10.350 


83.432 


1.036 


0.920 


1.078 


1.779 


9.084 


80.253 


1.042 


0.885 


2.646 


10.454 


9.763 


44.765 


1.053 


0.909 


6.594 


5.886 


14.379 


133.295 


1.036 


0.916 


1.045 


0.908 


7.086 


116.778 


1.040 


0.934 


1.578 


3.836 


10.524 


62.910 


1.046 


0.955 


3.114 


1.681 


14.278 


92.026 



Notes: 

All results based oa a=80 items and N=2000 respondents. 



1.001 


1.054 


0.928 


0.804 


1.790 


1.296 


1.011 


1.067 


1.0O8 


0.776 


10.183 


370.061 


1.012 


1.094 


2.600 


14.554 


11.918 


53.081 


1.054 


1.746 


8.387 


31.775 


11.527 


78.033 


1.019 


0.946 


1.000 


0.786 


6.685 


165.024 


1.021 


0.888 


1.736 


4.773 


9.705 


36.368 


1.030 


0.892 


3.554 


2.554 


14.892 


45.144 


1.017 


0.944 


0.999 


0.770 


4.004 


44.388 


1.017 


0.916 


1.199 


1.205 


10.082 


42.093 


1.022 


0.966 


1.677 


0.763 


14.166 


68.059 
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Table 5: 

Revised Modified Parallel Analysis (RMPA) of 20 tests: 

Correlations of eigenvalues of the observed and the expected jacknifed submatrices 



Rep r 



Ev 1 Ev 2 



Ev3 



Rep r p Ev 1 Ev 2 Ev 3 



B 
B 
B 
B 
B 
B 
B 
B 
B 
B 



0.0 
0.0 
0.0 
0.5 
0.5 
0.5 
0.7 
0.7 
0.7 



0 
10 
25 
50 
10 
25 
50 
10 
25 
50 



0.996 
0.995 
0.996 
0.981 
0.997 
0.998 
0.990 
0.997 
0.996 
0.997 



0.888 
-0.237 
-0.337 
-0.459 
•0.256 
-0.320 
-0.180 
0.863 
-0.311 
-0.206 



0.650 
0.603 
0.655 
-0.049 
0.299 
0.593 
0.310 
-0.111 
0.608 
0.508 



R 
R 
R 
R 
R 
R 
R 
R 
R 
R 



0.0 
0.0 
0.0 
0.5 
0.5 
0.5 
0.7 
0.7 
0.7 



0 
10 
25 
50 
10 
25 
50 
10 
25 
50 



0.976 
0.995 
0.996 
-0.641 
0.998 
0.996 
0.996 
0.998 
0.995 
0.996 



0.956 
0.963 
-0.414 
0.537 
0.968 
-0.320 
-0.007 
0.968 
0.350 
0.060 



Notes: 

All results based on n=80 items and N=2000 respondents. 
Ev = Eigenvalue 



0.195 

-0.147 

0.306 

0.321 

-0.129 

0.394 

0.218 

-0.111 

0.140 

0.127 
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Table 6: 



Revised Modified Parallel Analysis {RMPA) of 20 te^^^^^ 
Six detection thresholds based on the distribution of the standardized 
Weighted Gaps (SWGs) based on the ratio of the first observed and expected 
jacknifed eigenvalues 



Rep r 



0.0 



B 0.0 

B 0.0 

B 0.5 

B 0.5 

B 0.5 

B 0.7 

B 0.7 

B 0.7 

B Mean 



R 
R 
R 
R 
R 
R 



0.0 
0.0 
0.0 
0.5 
0.5 



R 0.5 
R 0.7 



0.7 
0.7 



R Mean 
Mean 



0 
10 
25 
50 
10 
25 
50 
10 
25 
50 



0 
10 
25 
50 
10 
25 
50 
10 
25 
50 



SWG 
Mean S.D. 



0.94 
0.91 
0.97 
0.84 
0.90 
0.96 
0.97 
0.98 
0.95 
0.89 



0.49 
0.58 
0.52 
0.56 
0.42 
0.53 
0.51 
0.50 
0.57 
0.52 



0.93 0.52 



0.95 
0.99 
0.94 
0.79 
0.99 
0.95 
0.93 
1.07 
0.92 
0.96 

0.95 

0.94 



0.44 
0.56 
0.54 
0.51 
0.48 
0.52 
0.53 
0.61 
0.45 
0.47 

0.51 

0.52 



1.89 
1.94 
1.96 
2.07 
1.61 
2.01 
1.84 
1.89 
2.13 
1.92 

1.93 

1.65 
1.85 
1.90 
1.77 
1.82 
2.03 
1.82 
2.16 
1.95 
1.85 

1.88 

1.90 



Threshold 



95% 



99 % 



Enp UChc Chcb 



2.42 
2.67 
2.54 
2.53 
2.15 
2.56 
2.50 
2.48 
2.65 
2.45 

2.49 

2.27 
2.66 
2.56 
2.30 
2.42 
2.52 
2.53 
2.90 
2.28 
2.38 

2.48 

2.49 



3.17 
3.54 
3.32 
3.37 
2.77 
3.36 
3.26 
3.23 
3.50 
3.23 

3.27 

2.93 
3.50 
3.37 
3.06 
3.14 
3.31 
3.33 
3.81 
2.95 
3.09 

3.25 

3.26 



Emp 



UChe Cheb 



2.38 
2.58 
2.87 
3.09 
1.92 
2.93 
2.26 
2.80 
2.66 
2.13 

2.56 

2.12 
3.40 
2.65 
2.47 
1.93 
2.24 
3.00 
2.87 
2.23 
2.03 

2.49 

2.53 



4.24 
4.81 
4.45 
4.59 
3.67 
4.52 
4.37 
4.31 
4.73 
4.35 



3.89 
4.72 
4.54 
4.16 
4.18 
4.44 
4.48 
5.12 
3.93 
4.12 

4.35 

4.38 



5.88 
6.76 
6.19 
6.46 
5.06 
6.30 
6.07 
5.97 
6.61 
6.08 



4.40 6.14 



5.36 
6.58 
6.33 
5.84 
5.78 
6.18 
6.25 
7.14 
5.43 
5.70 

6.06 

6.10 



Notes: 

All results based on n=80 items and N=2000 respondents. 
Emp = Empirical distribution 
UChe = Chcbyshcv bound assuming unimodality 
Chcb = Chcbyshcv bound 
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Table 7: 

Revised Modified Parallel Analysis (RMPA): 

Proportion of Standardized Weighted Gaps (SWGs) exceeding each of the six 
thresholds in ihe uncontaminated unidimensional test 









Threshold 










95 % 






99 % 




Eigenvalue 


Emp 


UChe 


Cheb 


Emp 


UChe 


Cheb 


I 
2 
3 


0.051 
0.177 
0.215 


0.000 
0.063 
0.108 


0.000 
0,019 
0.038 


0.013 
0.070 
0.120 


0.000 
0.006 
0.006 


0.000 
0.000 
0.000 


Mean 


0.148 


0.057 


0.019 


0.068 


0.004 


0.000 



Notes: 

All results based on n=80 items and N=2000 respondents. 
Emp = Empirical distribution 
UChe = Chebyshev bound assuming unimodality 
Cheb = Chebyshev bound 
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Table 8a: 

Revised Modified Parallel Analysis (RMPA) of 20 tests: 

Maximal Standardized Weighted Gap (SWG) and significance according to three 
types of thresholds (First eigenvalue) 













Significance* 


No. of items 


1 

Rep 


r 


P 


Gap 


Max(SWG) 


95% yy% 


Below 


Above 


B 
B 


- 


0 


0.00007800 


2.37665 


1 1 


40 


40 


0.0 


10 


0.00011536 


2,58303 


1 1 
1 1 


29 


51 


B 


0.0 


25 


0.00017863 


2.87133 


O 1 

Z I 


52 


28 


B 


0.0 


50 


0.00074733 


3.09437 




46 


34 


B 


0.5 


10 


0.00007026 


1.92493 


1 I 


42 


38 


B 


0.5 


25 


0.00023 166 


2.93434 




20 


60 ! 


B 


0.5 


50 


0.00011191 


2.25905 


1 1 

1 1 


49 


31 


B 


0.7 


10 


0.00011347 


z.oOlUJ 


9 1 


35 


45 


B 


0.7 


25 


0.00059257 


2.00J!) / 






4 


B 


0.7 


50 


0.00035381 


2. 12oiy 


1 1 
1 i 


76 


4 


R 




0 


0.00013665 


2.12450 


1 1 


30 


50 


R 


0.0 


10 


0.00083803 


3.39892 


2 1 


4 


76 


R 


0.0 


25 


0.00019422 


2.65083 


2 1 


22 


58 


R 


0.0 


50 


0.00513373 


2.47295 


2 1 


40 


40 


R 


0.5 


10 


0.00004417 


1.92817 


1 1 


33 


47 


R 


0.5 


25 


0.00010552 


2.24198 


1 1 


38 


42 


R 


0.5 


50 


0.00017510 


2.99795 


2 1 


28 


52 


R 


0.7 


10 


0.00010866 


2.87002 


1 1 


16 


64 


R 


0.7 


25 


0.00009185 


2.22529 


I 1 


33 


47 


R 


0.7 


50 


0.00017596 


2.02883 


1 1 


7 


73 



*Note: 

1 -> Max(SWG) > Empirical percentile 

2 -> Max(SWG) > Chcbyshev + unimodality 

3 -> Max(SWG) > Chcbyshev 
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Table 8b: 

Revised Modified Parallel Analysis (RMPA) of 20 tests: 

Maximal Standardized Weighted Gap (SWG) and significance according to three 
types of thresholds (Second eigenvalue) 













Significance* 


No. of items 


Reo 




n 

r 


Gap 


Majc(SWG) 


95% 


99% 


Below 


Above 


B 


- 


0 


0.002948 


4.6028 


3 


2 


13 


^T 

67 


B 


0.0 


10 


0.154001 


10,3842 


3 


3 


72 


8* 


B 


0.0 


25 


0.058056 


5.5693 


3 


2 


61 


19* 


B 


0.0 


50 


0.420034 


2.9129 


2 


0 


6 


74 


B 


0.5 


10 


0.061734 


7.9202 


3 


3 


72 


O Ik 


B 


0.5 


25 


0.020733 


3.9763 


3 


1 


62 


1 O 4( 


B 


0.5 


50 


0.016864 


2.7078 


2 


1 




jU 


B 


0.7 


10 


0.017799 


3,6832 


3 


1 


7o 


/ 


E 


0.7 


25 


0.017849 


4.2707 


3 


1 


64 


10 


B 


0.7 


50 


0.061186 


2.6536 


2 


I 


3 


77 


R 




0 


0.002074 


2.6308 


2 


1 


4 


76 


R 


0.0 


10 


0.001358 


3.1794 


2 


0 


10 


70 


R 


0.0 


25 


0.041428 


5.1730 


3 


2 


60 


20* 


R 


0.0 


50 


0.063916 


7.3824 


3 


3 


15 


65 


R 


0.5 


10 


0.002657 


2.7704 


2 


I 


4 


76 


R 


0.5 


25 


0.032784 


6.7929 


3 


3 


61 


19* 


R 


0.5 


50 


0.027717 


3.0870 


2 


1 


7 


73 


R 


0.7 


10 


0.000493 


2.7074 


1 


0 


23 


57 


R 


0.7 


25 


0.005675 


2.7842 


2 


1 


71 


9 


R 


0.7 


50 


O.0O5415 


2.7835 


2 


1 


19 


61 



1 -> Max(SWG) > Empirical percentile 

2 --> Max(SWG) > Chebyshev + unimodality 

3 -> Max(SWG) > Chebyshev 
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Table Sc: 

Revised Modified Parallel Analysis (RMPA) of 20 tests: 

Maximal Standardized Weighted Gap (SWG) and significance according to three 
types of thresholds (Third eigenvalue) 



Significance* No. of items 



Rep 


r 


P 


Gap 


Max(SWG) 


95% 


99% 


Below 


Above 1 

i 

1 


B 


- 


0 


0.197155 


5.0647 


3 


/ 




j 

i 

/O ! 


B 


0.0 


10 


0.407039 


3.5361 




1 


I 


7Q } 


B 


0.0 


25 


0.087976 


4.3483 


3 


I 


Q 

y 


71 1 

/ i j 


B 


0.0 


50 


0.065372 


3.5205 


3 


1 


1 1 


Oo 


B 


0.5 


10 


0.110290 


3.4420 


3 


I 


/J 


7 I 


B 


0.5 


25 


0.138565 


4.4914 


3 


I 


0 


7A i 


B 


0.5 


50 


0.237589 


5.5091 


3 


z 


Q 


71 ' 


B 


0.7 


10 


0.184939 


7.1549 


3 


3 


7 1 


Q + 


B 


0.7 


25 


0.140215 


5.1461 


3 


I 


Q 
O 


77 


B 


0.7 


50 


0.116452 


3.3573 


3 


I 


7 


73 


R 




0 


0.013257 


3.6044 


3 


1 


9 


71 


R 


0.0 


10 


0.419092 


9.4241 


3 


3 


72 


8* 


R 


0.0 


25 


0.115029 


4.1520 


3 


1 


10 


70 


R 


0.0 


50 


0.225673 


8.3301 


3 


3 


10 


70 j 


R 


0.5 


10 


0.287058 


10.3919 


3 


3 


72 


8* 


R 


0.5 


25 


0.118107 


5.3490 


3 


2 


12 


68 


R 


0.5 


50 


0.464085 


4.3802 


3 


1 


3 


11 


R 


0.7 


10 


0.111553 


6.4281 


3 


2 


72 


8* 


R 


0.7 


25 


0.072746 


3.5625 


3 


1 


11 


69 


R 


0.7 


50 


0.272261 


4.7821 


3 


2 


7 


73 



1 Max(SWG) > Empirical percentile 

2 -> Max(SWO) > Chebyshev + unimodality 

3 -> Max(SWG) > Chebyshev 
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Table 9: 

Revised Modified Parallel Analysis (RMPA) of 10 short tests: 

Total number of items eliminated and accuracy of the elimination procedure 



Rep 

i 


r 


P 


Total 


Items eliminated 

% of % of 
"hits" "false alarms" 


Significant 
Eigenvalue 


B 


0.0 


10 


8 


100 


0 


2 


B 


0.5 


10 


8 


100 


0 


2 


B 


0.7 


10 


9 




1 




R 


0.0 


10 


8 


100 


0 


3 


R 


0.5 


10 


8 


100 


U 


o 


R 


0.7 


10 


8 


100 


0 


3 


Mean 






8.2 


100 


0.2 




B 


0.0 


25 


19 


95 


0 


2 


*r> 
D 




25 


18 


90 


0 


2 


R 


0.0 


25 


20 


100 


0 


2 


R 


0.5 


25 


19 


95 


0 


2 


Mean 






19 


95 


0 




Mean 








98 


0.1 





Note: 

Tests shortened by 99% criterion 

♦ These tests shortened by a 95% criterion 
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Table 10: 

Modified Parallel Analysis (MPA) of 10 short tests: 

The first three eigenvalues for the observed and expected matrices, and their ratios 



Eigenvalue 1 



Eigenvalue 2 



Eigenvalue 3 



Repr p Exp Obs Obs/Exp Exp Obs Obs/Exp Exp Obs Obs/Expi 



B 
B 
B 
B 
B 

R 
R 
R 
R 
R 



0.0 
0.0 
0.5 
0.5 
0.7 

0.0 
0.0 
0.5 
0.5 
0.7 



10 
25 
10 
25 
10 

10 
25 
10 
25 
10 



21.86 
17.89 
21.95 
18.00 
21.49 

23.35 
19.17 
22.79 
19.27 
22.83 



22.71 
18.78 
22.71 
18.85 
22.27 

23.84 
19.79 
23.21 
19.73 
23.21 



1.04 
1.05 
1.03 
1.05 
1.04 

1.02 
1.03 
1.02 
1.02 
1.02 



1.62 
1.42 
1.65 
1.40 
1.58 

2.84 
2.37 
2.70 
2.33 
2.73 



1.65 
1.48 
1.65 
1.49 
1.61 

2.85 
2.30 
2.71 
2.45 
2.71 



1.02 
1.05 
1.00 
1.06 
1.01 

1.01 
0.98 
1.00 
1.05 
0.99 



0.17 
0.14 
0.18 
0.15 
0.17 

0.25 
0.19 
0.28 
0.24 
0.28 



0.62 
0.56 
0.62 
0.58 
0.62 

0.54 
0.50 
0.58 
0.53 
0.58 



3.64 
3.86 
3.51 
3.89 
3.63 

2.19 
2.63 
2.10 
2.21 
2.04 



Notes: ^ . . 

All results based on N=2000 respondents, and various number ot items. 
Exp = Derived from matrix of expected correlations 
Obs = Derived from matrix of observed correlations. 



Table 11: 



Revised Modified Parallel Analysis (RMPA) of 10 short tests: 

Ratio of means and variances of eigenvalues of the jacknifed submatrices 

{Ratio = observed / expected) 









Eigenvalue 1 


Eigenvalue 2 


1 

1 

Eigenvalue 3 


Rep 


r 


P 


Mean 


Var 


Mean 


Var 


Mean 


Var 


B 


0.0 


10 


1.039 


0.928 


1.023 


1.066 


3.660 


5.473 


B 


0.0 


25 


1.049 


0.923 


1.046 


1.032 


3.889 


4.006 


B 


0.5 


10 


1.035 


0.922 


1.005 


0.983 


3.526 


5.112 


B 


0.5 


25 


1.047 


0 959 


1.064 


1.065 


3.913 


6.538 


B 


0.7 


10 


1.036 


0.919 


1.015 


1.072 


3.641 


5.785 


R 


0.0 


10 


1.021 


0.975 


1.005 


0.816 


2.202 


1.152 


R 


0.0 


25 


1.033 


0.947 


0.976 


0.760 


2.648 


1.670 


R 


0.5 


10 


1.018 


0.960 


1.001 


0.785 


2.107 


1.299 


R 


0.5 


25 


1.024 


0.997 


1.054 


0.863 


2.227 


1.418 


R 


0.7 


10 


1.017 


0.949 


0.991 


0.772 


2.047 


1.102 
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